Storage system having sas as its backend communication standard

ABSTRACT

Each communication path between controllers and a plurality of storage devices has a plurality of expanders coupled in series. In order to shorten the time during which the communication path is not used for I/O, either (A) the length of time for which I/O suppression is set for the communication path is shortened, or (B) the overall time it takes for processing other than I/O processing is shortened. In the (A), a determination as to whether or not the coupling between the expanders has been disconnected is made for the I/O-suppressed communication path, and in a case where the result of this determination is negative, a discover process is carried out after releasing the I/O suppression with respect to this communication path. In the (B), the number of command issue times of updating routing control information of the expander is reduced.

TECHNICAL FIELD

The present invention relates to a storage system that has SAS (SerialAttached SCSI) as its backend communication standard.

BACKGROUND ART

The storage system 101 shown in FIG. 1, for example, is known as astorage system that has SAS as its backend (between the controller andthe storage device) communication standard. In the explanation thatfollows, a plurality of elements of the same type may be distinguishedby using an element identification number. For example, whendistinguishing between a plurality of element As, these elements may bedescribed as “A #x”. “A #x” signifies that the identification number ofelement A is x. The “x” is an integer of equal to or larger than 0.

The storage system 101 comprises a plurality of storage devices 151, aredundant controller 131 (Controllers #0 and #1), a first communicationpath 161 to which the controller #0 and the plurality of storage devices151 are coupled, and a second communication path 163 to which thecontroller #1 and the plurality of storage devices 151 are coupled.

Each controller 131 is coupled to a host 103 via a communication network105, and comprises a SAS controller 141 that carries out communicationsin accordance with the SAS protocol. The controller #0 (#1) receives anI/O request from the host 103. The controller #0 (#1) issues via the SAScontroller #0 (#1) an I/O (Input/Output) command targeted at the storagedevice 151 identified based on this I/O request. Or, the controller #0(#1) transfers the I/O request from the host 103 to the other controller#1 (#0), and the controller #1 (#0) issues via the SAS controller #1(#0) an I/O command targeted at the storage device 151 identified basedon this I/O request. The second communication path 163 is the redundantroute of the first communication path 161. For this reason, each storagedevice 151 is able to be accessed by way of either the first or secondcommunication path 163.

Each communication path comprises a plurality of SAS expanders(hereinafter, expander) 143 coupled in series. The first initial-stageexpander 143 of the plurality of expanders 143 is coupled to thecontroller 131. Therefore, the controller 131 and the plurality ofexpanders 143 are in series. The controller 131 and the expander 143,and the expander 143 and the expander 143 are coupled by way of physicallinks (physical wires).

In a storage system of this type, a discover process is carried out inthe backend. In the discover process, a command (a SMP command) thatconforms to the SMP (Serial Management Protocol) is issued from thecontroller 131. The SMP command passes through the same physical link asthe physical link that is capable of being used in an I/O (Input/Output)to/from the storage device 151. In accordance with SMP, the physicallink through which this command passes is occupied from the time thecommand is issued from the controller 131 (initiator) to the targetdevice (either the expander 143 or the storage device 151) until aresponse is returned to the controller 131. Therefore, this physicallink is also occupied while the command is being processed. For thisreason, the greater the number of SMP commands, the more difficult it isto use the physical link for I/O, and I/O performance is likely todeteriorate as a result.

Patent Literature 1 discloses SSP (Serial SCSI Protocol) as a type ofprotocol that does not occupy the physical link while the command isbeing processed. Issuing an SSP command instead of the SMP command inthe discover process makes it possible to reduce deterioration in I/Operformance.

CITATION LIST

[Patent Literature]

[PTL 1]

-   Japanese patent application Laid-open Publication No. 2008-242872

SUMMARY OF INVENTION Technical Problem

However, regardless of the type of command issued in the discoverprocess (for example, regardless of whether this command is a SMPcommand or a SSP command), it is not always preferable to issue an I/Ocommand to the storage device during the discover process. There arecases where the controller 131 receives a lot of error responses(responses denoting errors) when an I/O command is issued to the storagedevice during the discover process.

One example of such a case is that shown in FIG. 2 where all thephysical links between the expander #01 and the expander #02 aredisconnected (for example, a case in which the cables between theexpander #01 and the expander #02 become unplugged). In this case, thefact that all the physical links between the expander #1 and theexpander #02 have been disconnected cannot be detected without carryingout a discover process.

The controller #0 issues an I/O command prior to detecting this fact.However, in a case where the target of this I/O command is a storagedevice coupled to an expander that comes after the expander #01, thecontroller #0 receives an error response with respect to the I/Ocommand. Therefore, in a case where the controller #0 issues a lot ofI/O commands targeted at the storage device coupled to the expander thatcomes after the expander #01, it receives a lot of error responses.

In a case like this, it is preferable that, as shown in FIG. 3, the I/Ocommand go by way of the redundant communication path (the secondcommunication path 163) rather than the communication path (firstcommunication path 161) being used in the discover process.

However, in a case like this where the I/O command is issued to thestorage device 151 via a communication path other than the communicationpath being used in the discover process, the communication path beingused in the discover process is not used for I/O. For this reason, thefollowing problems occur.

-   -   (1) I/O performance deteriorates.    -   (2) Redundancy deteriorates. Specifically, in a case where the        communication path is p-plexed and there are q number of        communication paths being used in the discover process,        redundancy becomes (p−q). Therefore, when redundancy is 2,        redundancy becomes 0 in a case where one communication path is        being used in the discover process.

Accordingly, an object of the present invention is to shorten the timeperiod that a communication path is not used for I/O.

Solution to Problem

There is a first communication path (a communication path that conformsto SAS) coupled to a first controller and to a plurality of storagedevices, and a second communication path (a communication path thatconforms to SAS) coupled to a second controller and to the plurality ofstorage devices. Each communication path comprises a plurality ofexpanders that are coupled in series. Each expander comprises aplurality of phy, and the plurality of phy are coupled to the phy, whichare coupled to another expander, and are coupled to the storage device.Each expander comprises routing control information that is used inrouting. The routing control information may comprise SAS addressinformation, which denotes the SAS address of the device that isindirectly coupled to each phy.

According to a first aspect, I/O suppression is set for the firstcommunication path, but the length of time for I/O suppression isshortened. Specifically, the first controller determines whether or notthe coupling between the expander and the expander (or the couplingbetween the first controller and the expander) has been disconnected. Ina case where the result of this determination is negative, afterreleasing the I/O suppression with respect to the first communicationpath, the first controller carries out an investigation process (forexample, a discover process) that includes a process that issues withrespect to at least one expander on the first communication path acommand for acquiring information related to the device that is directlycoupled to this expander. In accordance with this, an I/O commandtargeted at the storage device is sent by way of the first communicationpath even when the investigation process is in progress for the firstcommunication path. A case in which the result of the abovedetermination is negative, for example, is one where the couplingbetween the expander and the storage device has been disconnected.However, the impact of an error response sent in a case where thecoupling between the expander and the storage device has beendisconnected is small since this error only pertains to the I/O commandtargeted at this storage device.

According to a second aspect, the overall time that it takes for aprocess (for example, the discover process) other than an I/O processwith respect to the storage device is shortened. Since the number ofstorage devices mounted to the storage system will most likely increasein the future, the shortening of the overall time that it takes for aprocess other than the I/O process is effective. Specifically, accordingto the second aspect, the number of commands (typically the CRI(Configure Route Information) command) for updating the routing controlinformation of the expander can be reduced. More specifically, forexample, at least one of the following (1) through (3) is employed.

-   -   (1) A first original CRI command is issued for setting the same        SAS address information for two or more phy coupled in a wide        link. In accordance with this, SAS address information can be        set in the routing control information for two or more phy using        a single CRI command. That is, there is no need to issue a CRI        command for each phy to set the SAS address information.    -   (2) Even in a case where a link-down is detected for a storage        device, the issuing of a CRI command is skipped when a        prescribed condition has been satisfied for this link-down.    -   (3) In a case where an expander is coupled to a controller after        the coupling between this controller and the expander has been        disconnected, this controller issues a second original CRI        command to this expander. The expander that receives the second        original CRI command deletes all SAS address information from        the routing control information. That is, there is no need to        issue a CRI command for deleting the SAS address information for        each piece of SAS address information.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 shows an example of the configuration of a storage system inwhich SAS is the expander communication standard.

FIG. 2 is a diagram for illustrating the problem that occurs when thecoupling between an expander and an expander is disconnected.

FIG. 3 shows that an I/O is performed via a second communication pathwhile a discover process is being carried out on a first communicationpath.

FIG. 4 shows the configuration of a storage system related to anembodiment of the present invention.

FIG. 5 shows the configuration of a controller 431.

FIG. 6 shows the configuration of an expander 443.

FIG. 7A shows the configuration of an expander route table 611.

FIG. 7B shows the configuration of the communication path correspondingto the table 611 shown in FIG. 7A.

FIG. 8 shows the configuration of a device management table 511.

FIG. 9 shows the processing flow of a discover process.

FIG. 10A shows an example of a case in which it is preferable that I/Osuppression be maintained. FIG. 10B shows an example of a case in whichit is preferable that I/O suppression be released.

FIG. 11 shows the processing flow of a process related to Aspect 1.

FIG. 12 shows an overview of Aspect 2-1.

FIG. 13 shows the configuration of a wide link bundle command.

FIG. 14 shows the configuration of an ordinary CRI (Configure RouteInformation) command.

FIG. 15 shows an example of a phy bitmap.

FIG. 16 shows the processing flow of a process (a process performed bythe controller 431) that conforms to the Aspect 2-1.

FIG. 17 shows the processing flow of a process performed by the expander443.

FIG. 18A shows a problem that may occur in a storage system in which anAspect 2-2 is not applied. FIG. 18B shows that “invalid” is set in theexpander route table for all phy that are indirectly coupled to PDEV:K(the storage device at SAS address “K”) for which a link-down occurred.FIG. 18C shows that “valid” is set in the expander route table for allphy that are indirectly coupled to PDEV:K for which a link-up occurred.

FIG. 19 shows the processing flow of a process (a process performed bythe controller 431) that conforms to the Aspect 2-2.

FIG. 20 shows the processing flow of a failure check process (S1905 ofFIG. 19).

FIG. 21 shows the processing flow of a link-down check process (S1907 ofFIG. 19).

FIG. 22A shows an example of a configuration in which two expanders #011and #012 are directly coupled to one expander #00. FIG. 22B shows thatthe coupling between the controller #0 and the expander #00 isdisconnected.

FIG. 23A shows that the expanders that are coupled to phy #1 and phy #2of the expander #00 were switched subsequent to the disconnection shownin FIG. 22B. FIG. 23B shows that the controller #0 and the expander #00were coupled subsequent to the switch shown in FIG. 23A.

FIG. 24A shows the result (the expander route table 611 of the expander#00) of carrying out the discover process subsequent to the couplingshown in FIG. 23B. FIG. 24B shows that it becomes impossible to detectthe transfer-destination of the routing based on the table 611 shown inFIG. 24A.

FIG. 25A shows the batch deletion of all SAS address information fromthe expander route table 611 of the expander #00 prior to the discoverprocess subsequent to the coupling shown in FIG. 23B. FIG. 25B shows theresult (the expander route table 611 of the expander #00) of carryingout the discover process subsequent to the batch delete shown in FIG.25A.

FIG. 26 shows an example of the configuration of a storage system inwhich transfer-destination detection could become impossible.

FIG. 27 shows an overview of an Aspect 2-3.

FIG. 28 shows the configuration of a batch delete command.

FIG. 29 shows the processing flow of a process (a process performed bythe controller 431) that conforms to the Aspect 2-3.

DESCRIPTION OF EMBODIMENTS

One example of the present invention will be explained below byreferring to the drawings.

In the following explanation, various types of information are explainedusing the expression “kkk table”, but these various types of informationmay be expressed using a data structure other than a table. To show thatthe information is not dependent on the data structure, “kkk table” maybe called “kkk information”.

Further, a number will be used as information for identifying varioustargets in the following explanation, but other types of identificationinformation (for example, an identifier comprising a letter and symbol)may also be used.

In the following explanation, there may be instances when “program” isused as the subject in explaining a process, but since a prescribedprocess is performed in accordance with a program being executed by aprocessor (for example, a CPU (Central Processing Unit)) while using astorage resource (for example, a memory) and/or a communicationinterface device (for example, a communication port) as needed, thesubject of the process may also be the processor. A process that isexplained having the program as the subject may be a process that isperformed by a storage system, a controller or an expander. Theprocessor may comprise a hardware circuit for processing either all or aportion of a process performed by the processor. A computer program maybe installed in respective computers from a program source. The programsource, for example, may be a program delivery server or a storagemedium.

In the explanation that follows, a plurality of elements of the sametype may be distinguished by using an element identification number. Forexample, when distinguishing between a plurality of element As, theseelements may be referred to as “A #x”. “A #x” signifies that theidentification number of element A is x. The “x” is an integer of equalto or larger than 0.

In the following explanation, a SAS expander will simply be called an“expander”. The expander is a switching device.

In the following explanation, the controller side may be called the“initial stage”, and the side opposite the controller side may be calledthe “subsequent stage”.

In the following explanation, the coupling of a device (a storage deviceor an expander) to a phy without a one-phy separation will be referredto as “directly coupled”, and the coupling of a device to a phy by wayof one or more other phys will be referred to as “indirectly coupled”.

In the following explanation, a device (a storage device or anexpander), which is directly coupled to a phy, will be referred to as a“direct device”, and a device, which is indirectly coupled to a phy,will be referred to as an “indirect device”.

In the following explanation, the inability to communicate via a phy toa device that is directly coupled to this phy will be called“link-down”, and the recovery of this link-down (that is, the ability tocommunicate) will be called “link-up”.

In the following explanation, the inability of an expander #x (or thecontroller) to communicate with an expander # (x+1) (thesubsequent-stage expander after the expander #x) may be referred to as“the expander #x (or the controller) and the expander # (x+1) have beendisconnected”. A disconnection, for example, occurs as a result of thecable for coupling the expander #x (or the controller) and the expander# (x+1) being severed or unplugged. In a case where the expander #x (orthe controller) and the expander # (x+1) are coupled together using awide link, and, in addition, a link-down occurs for all the phy that arecoupled to the wide link, this may be described as a “disconnection”having occurred.

FIG. 4 shows the configuration of a storage system related to anembodiment of the present invention.

A storage system 401 is configured from a primary chassis 411 and aplurality of additional chassis 413. An additional chassis 413 iscoupled to the primary chassis 411, and one or more additional chassis413 are coupled in series to the subsequent-stage of this additionalchassis 413.

The primary chassis 411 comprises duplicated controllers 431(controllers #0 and #1). The primary chassis 411 may also comprise astorage device (hereinafter, a PDEV) 451.

The controllers #0 and #1 are coupled to a host 403 by way of acommunication network 405. The communication network 405, for example,is a SAN (Storage Area Network). The host 403 may be one or morephysical computers or one or more virtual computers. The host 403 issuesan I/O request.

The controller 431 comprises a SAS controller 441.

The additional chassis 413 comprises duplicated enclosures 433 and aplurality of PDEV 451. Each enclosure 433 comprises an expander 443. ThePDEV 451, for example, is either a HDD (Hard Disk Drive) equipped witheither an SATA (Serial Attached SCSI) or a SAS interface, or a SSD(Solid State Drive).

There is a first communication path 461, which is coupled to thecontroller #0 and the plurality of PDEV 451, and a second communicationpath 463, which is coupled to the controller #1 and the plurality ofPDEV 451. The second communication path 463 is the redundant path of thefirst communication path 461. For this reason, it is possible to accesseach PDEV 451 via either the first or second communication paths 463.

Each communication path comprises a plurality of expanders 443 that arecoupled in series. The first initial-stage expander 443 is coupled tothe controller 431. Therefore, the controller 431 and the plurality ofexpanders 443 are coupled in series. The controller 431 and the expander443, and the expander 443 and the expander 443 are coupled using a widelink 461. The wide link 461 is configured from two or more parallelphysical links. In place of the wide link 461, the controller 431 andthe expander 443 and/or the expander 443 and the expander 443 may becoupled using a physical link that does not configure a wide link.

The controller #0 (#1) receives an I/O request from the host 403. Thecontroller #0 (#1) identifies the I/O-destination PDEV 451 on the basisof the received I/O request. The controller #0 (#1) issues an I/Ocommand targeted at the identified PDEV 451 by way of a SAS controller#0 (#1). This I/O command passes through the expander #0 n (#1 n) (wheren is an integer of equal to or larger than 0) in the first communicationpath 461 (second communication path 463) coupled to the controller #0(#1).

FIG. 5 shows the configuration of the controller 431.

The controller 431 comprises a RAID controller 571 and a backend I/F521.

The RAID controller 571 receives an I/O request (either a write requestor a read request) from the host 403, identifies the PDEV 451 based onthe logical volume identified from this I/O request, creates an I/Ocommand targeted at the identified PDEV 451, and sends this I/O commandto the SAS controller 441. The RAID controller 571 comprises a host I/F507, a cache memory 503, a timer 505, a memory 509, and a processor (forexample, a CPU) 501 that is coupled thereto.

The host I/F 507 is an interface device for communicating with the host403. The host I/F 507 receives the I/O request from the host 403.

The cache memory 503 temporarily stores data that is either written toor read from the PDEV 451 in accordance with the I/O request from thehost 403.

The timer 505 measures the current time.

The memory 509 stores information and computer programs, for example, aninitial save time table 510, a device management table 511, a link-downcount table 512, a RAID control program 513, an I/O issuing program 514,a device command issuing program 515, a failure processing program 516,and a device control program 517. The CPU 501 executes the programs 513through 517. Table 510 through 512 will be explained further below.

The RAID control program 513 performs a RAID operation. Specifically,for example, a RAID group is configured from two or more PDEV 451, andthe RAID control program 513 performs a RAID operation that utilizesdata in accordance with the RAID level of the RAID group that is thewrite-destination of this data.

The I/O issuing program 514 identifies a data write-destination PDEV 451based on the result of the RAID operation, creates a write command thattargets the identified PDEV 451, and issues this created write command.

The device command issuing program 515 issues a command (for example, anoriginal command such as a wide link bundle command, which will bedescribed below) that differs from an I/O command.

The failure processing command 516 performs processing related to afailure check (for example, counting the number of errors, monitoringthe time, and making various determinations).

The device control program 517 updates the device management table 511based on a response to the command issued by the device command issuingprogram 515.

The backend I/F 521 is an interface device for carrying outcommunications that conform to SAS. The backend I/F 521 comprises theSAS controller 441.

The SAS controller 441 is able to control the communication protocolconversion between the expander 443 and the RAID controller 517. The SAScontroller 441, for example, comprises a wide port. The wide port is aport configured from two or more physical ports, and is coupled to thefirst initial-stage expander 443 via the wide link. The SAS controller441 may comprise a plurality of wide ports, or may comprise a physicalport that does not configure a wide port. A plurality of expanders 443may be coupled to the SAS controller 441.

The controller 431 (for example, #0) is coupled to the other controller431 (for example, #1) by way of the host I/F 507, the backend I/F 521 ora communication interface that differs from these (not shown in thedrawing). In accordance with this, the controller 431 is able totransfer either the I/O request from the host 403 or the I/O commandtargeted at the PDEV 451 to the other controller 431.

FIG. 6 shows the configuration of the expander 443.

The expander 443, for example, is a LSI (Large Scale Integration). Theexpander 443 comprises a device I/F 601, a switching circuit 603, amemory 607, and a processor (for example, a CPU) 605.

The device I/F 601 is an interface device for a device. As used here, a“device” is either a PDEV 451 or another expander 443. The device I/F601 is configured from a plurality of phys (physical ports) 621. Theplurality of phys 621 comprise a phy 621 to which a PDEV 451 is directlycoupled, and a phy 621 to which another expander 443 is directlycoupled. In a case where the expander 443 is the first initial-stageexpander 443, the plurality of phys 621 comprise the phy to which thecontroller 431 is directly coupled.

The switching circuit 603 is a hardware circuit that performs routing.The plurality of phys 621 and the processor 605 are coupled to theswitching circuit 603. The switching circuit 603 comprises a memory 605,and the memory 605 stores an expander route table 611. Routing iscarried out on the basis of this expander route table 611.

The memory 607 stores information and computer programs, for example, adevice I/F information 623 and firmware 631.

The device I/F information 623 comprises status information for each phy621 (for example, “Ready”, which signifies that communication ispossible, and “Not-ready”, which signifies that communication is notpossible).

The firmware 631 comprises a device I/F control program 641, a deviceI/F monitoring program 642, a broadcast issuing program 643, and acommand processing program 644.

The device I/F control program 641 controls the valid/invalid of thephy.

The device I/F monitoring program 642 detects the status of each phy bypolling each phy. The device I/F monitoring program 642 writesinformation denoting the status detected for the phy to the device I/Finformation 623. Further, the device I/F monitoring program 642 boots upthe broadcast issuing program 643 when the detected status signifies anerror.

The broadcast issuing program 643 broadcasts error information. That is,error information is outputted to both the initial-stage side and thesubsequent-stage side. The error information that is outputted to theinitial-stage side eventually reaches the controller 443. Furthermore,the broadcast error information, for example, is BP (SES). The BP(Broadcast Primitive) is an information frame that is supported by theSAS standard. The BP, for example, includes BP (SES) (SES is theabbreviation for “SCSI Enclosure Service”) and BP (CHANGE). The BP(SES), for example, is the information frame that is created and issuedin a case where a change has been detected in the status of themonitoring targeted device. The BP (CHANGE), for example, is theinformation frame that is created and issued in a case where a change inthe topology managed by the expander 443 (for example, the addition orremoval of a PDEV 451) has been detected.

The command processing program 644 analyzes the command received fromthe controller 431, identifies the command type, and processes thiscommand in accordance with the identified type of command.

FIG. 7A shows the configuration of the expander route table 611.

The table 611 comprises SAS address information denoting the SAS addressof a device (either a PDEV or an expander) that is indirectly coupled toa phy for each phy of the expander (the expander comprising this table611) 443, and status information denoting whether this device is validor invalid.

The table 611 shown in FIG. 7A, for example, is for expander #00 anddenotes the configuration shown in FIG. 7B. Furthermore, theidentification number of the expander 443 is a serial number, whichincreases by one the later in the subsequent stage the expander 443comes. That is, the subsequent-stage expander that is directly coupledto expander #x is expander #(x+1), and the initial-stage expander thatis directly coupled to expander #x is expander #(x−1).

As shown in FIG. 7B, the PDEV of SAS address “A” (hereinafter, PDEV:A)and PDEV:B are directly coupled to the expander #01, and PDEV:C isdirectly coupled to the expander #02.

According to the configuration shown in FIG. 7B, all of PDEV:A, PDEV:Band PDEV:C are accessible via any of the phys #0 through #3 of theexpander #00. For this reason, the SAS address information “A”, “B” and“C” of the PDEV:A, the PDEV:B and the PDEV:C are set with respect to thephys #0 and #1 in the expander route table 611 of the expander #00 asshown in FIG. 7A. Although omitted from the drawing, the SAS addressinformation “A”, “B” and “C” of the PDEV:A, the PDEV:B and the PDEV:Care also set with respect to the phys #02 and #03 of the expander #00.

Also, although not shown in the drawing, each expander 443 may comprisea direct device table in addition to the expander route table 611. Thedirect device table denotes the SAS address information of a device thatis directly coupled to a phy of the expander that comprises this table.Upon receiving an I/O command comprising SAS address information, theexpander 443, for example, is able to determine whether SAS addressinformation that is identical to this SAS address information is in theexpander route table 611 or the direct device table. In a case where theidentical SAS address information is in the expander route table 611,the expander 443 is able to select one phy number, from among one ormore phy numbers, that correspond to this identical SAS addressinformation, and output the above-mentioned received I/O command fromthe phy of the selected phy number. Alternatively, in a case where theidentical SAS address information is in the direct device table, theexpander 443 is able to output the above-mentioned received I/O commandfrom the phy of the selected phy number that corresponds to thisidentical SAS address information.

FIG. 8 shows the configuration of the device management table 511.

The controller 431 comprises the table 511 (Refer to FIG. 5). The table511 comprises the following information for each expander,

-   -   (8-1) information denoting either (Wide), i.e., that the phy is        coupled to a physical link that configures a wide link, or        (Narrow), i.e., that the phy is coupled to a physical link that        does not configure a wide link, for each phy, and    -   (8-2) status information denoting normal or abnormal for each        phy. According to the example of FIG. 8, it is clear that        physical links, which configure the wide link #0, are coupled to        the phys #0, #1 and #2 of the expander #00 (Refer to FIG. 7B),        and that all of the phys #0, #1 and #2 are normal.

This FIG. 8 shows the table 511 of the controller #0. The table 511 ofthe controller #1 comprises the above-mentioned information (8-1) and(8-2) for the expander #1 n (where n is an integer equal to or largerthan 0).

FIG. 9 shows the processing flow of the discover process.

The discover process is performed for each communication path 461, 463.The discover process is carried out one at a time in order from anexpander on the initial-stage side to an expander on thesubsequent-stage side. That is, the discover process target shifts onesubsequent-stage expander at a time.

The following three types of commands are issued in the discoverprocess,

-   -   (1) RG (Report General) command,    -   (2) Discover command, and    -   (3) CRI (Configure Route Information) command.

All of the commands (1) through (3) are SMP commands.

The RG command is issued to a target expander. The RG command is issuedfor acquiring information (for example, the number of phys and theidentification number of a phy) related to the phys of the targetexpander. The RG command is issued for each expander.

The discover command is issued to a target expander. The discovercommand is issued for acquiring information (for example, the SASaddress of a PDEV) related to a device (either an expander or a PDEV)that is directly coupled to a phy of the target expander. The discovercommand is issued for each phy of this expander.

The CRI command is issued to an expander that comes earlier in theinitial stage than the target expander. The CRI command is issued forsetting information in the expander route table 611. The CRI command isissued to each expander that comes earlier in the initial stage than thetarget expander, and, in addition, is issued for each phy of eachexpander.

To make the following explanation easier to understand, it is supposedthat the expander #01 in the configuration shown in FIG. 7B is thetarget of the discover process.

The controller #00 first issues the RG command to the expander #01(S901). The expander #01 (the command processing program) receives theRG command, and returns a response comprising information related to thephy to the controller #00. The phy-related information comprises the phynumbers of all the phys in the expander #01. The controller #00 is ableto acquire the phy numbers of all the phys in the expander #01 from theresponse to the RG command.

Next, the controller #00 issues the discover command comprising the phynumber to the expander #01 (S902). The discover command is issued foreach phy of the expander #01. The expander #01 receives the discovercommand and returns to the controller #00 a response comprising the SASaddress information of the device that is directly coupled to the phy ofthe phy number of the received discover command. For example, theexpander #01, upon receiving the discover command comprising the phynumber “7”, returns a response comprising the SAS address information“A” to the controller #00. The controller #00 is able to acquire fromthe response to the discover command the SAS address information of thedevice that is directly coupled to the phy of the phy number of thisdiscover command.

Next, the controller #00 issues the CRI command to all the expanders #00that come earlier in the initial stage than the expander #01 (S903).This CRI command comprises the number of the phy of the expander that isthe destination of this CRI command, and the SAS address information(the SAS address information acquired from the response to the discovercommand) of the device that is indirectly coupled to this phy. Forexample, in a case where the destination of the CRI command is theexpander #00, the CRI command, for example, comprises the phy number“0”, the SAS address information “A”, and the index number “0”. Theexpander #00 (the command processing program), in accordance with thisCRI command, sets the SAS address information “A” in the locationcorresponding to the phy number “0” and the index number “0” in theexpander route table 611 (“Valid” may also be set as the statusinformation at this time). In accordance with this, it becomes possiblefor a command targeted at PDEV #A to go through the phy #0 of theexpander #00. Furthermore, the CRI command comprising the SAS addressinformation “A” and the index number “0” is also issued to each of thephy numbers “1” through “3”.

When the discover process (S901 through S903) with respect to the targetexpander #01 has ended, the controller #0 determines whether or not anexpander exists in the subsequent stage of the expander #01 (S904). Thisdetermination, for example, may be made based on the response to thediscover command. In a case where the result of the determination inS904 is negative, processing ends because the target expander is thelast subsequent-stage expander. Alternatively, in a case where theresult of the determination in S904 is affirmative, the target expanderbecomes the expander #02 that comes after the expander #01, and thediscover process is performed for the expander #02. In this discoverprocess, the CRI command is issued to the expanders #00 and #01 of theinitial stage with respect to the expander #02.

Based on the preceding explanation, the later in the subsequent stagethe target expander of the discover process exists, the more time thediscover process will take. This is because most of the expanders are inthe initial stage with respect to the target expander, thereforeresulting in more CRI commands being issued.

This embodiment makes it possible to shorten the time that acommunication path is unable to be used for I/O in accordance with atleast one the following Aspects 1 and 2.

-   -   (Aspect 1) The length of time of I/O suppression is shortened.    -   (Aspect 2) The number of CRI commands that are issued is        reduced. Specifically, this aspect is as follows.    -   (Aspect 2-1) A first original CRI command is defined for        collectively setting SAS address information in the expander        route table 611. The first original CRI command will be called        the “wide link bundle command” hereinbelow. A CRI command that        is issued in the discover process may be deemed a wide link        bundle command as appropriate. A single wide link bundle command        is used to collectively set SAS address information in the        expander route table 611 for all of two or more phys that are        coupled to one wide link. In accordance with this, it is not        necessary to issue a CRI command for each phy. Therefore, it is        possible to shorten the time required for the discover process.    -   (Aspect 2-2) The issuing of the CRI command for setting the        status information “invalid” in the expander route table 611 is        skipped as deemed appropriate. This reduces the number of CRI        commands issued.    -   (Aspect 2-3) A second original CRI command is defined for        collectively deleting the SAS address information from the        expander route table 611. The second original CRI command will        be called the “batch delete command” hereinbelow. A single batch        delete command is used to delete all the SAS address information        in the expander route table 611 from this table 611. This does        away with the need to issue the CRI command for deleting SAS        address information for each piece of SAS address information.

Each aspect will be explained in detail hereinbelow.

<Aspect 1: Shortens I/O Suppression Time>

Upon detecting an error of some sort, the expander 443 broadcasts errorinformation (BP (SES)). Error information that has been outputted to theinitial-stage side reaches the controller 431 by going one at a timethrough one or more expanders 443, which are in the stage furtherforward than the expander 443 that detected the error. The controller431, upon receiving the error information, sets I/O suppression for thecommunication path through which this error information passed. Thecontroller 431 does not output an I/O command to the communication paththat has been set to I/O suppression.

As shown in FIG. 10A, it is supposed that the coupling between theexpander #x (for example, #00) and the next expander # (x+1) (forexample, #01) has been disconnected. In a case like this, the controller(for example, #0) will receive an error response (a response denoting anerror) no matter which PDEV directly coupled to an expander later thanthe expander # (x+1) is targeted when it issues the I/O command. This isbecause the I/O command cannot be transmitted to an expander that comeslater in the subsequent stage than the expander #x.

Alternatively, as shown in FIG. 10B, in a case where it is supposed thatthe link to the PDEV (for example, PDEV: I) that is directly coupled tothe expander #x (for example, #02) has gone down, the only errorresponse received is that with respect to the I/O command that targetedthe PDEV:I. For this reason, the impact on the storage system as a wholeis minimal.

Accordingly, in this embodiment, the controller 431 checks beforehandonly whether or not the coupling between the expander #x and the nextexpander # (x+1) has been disconnected. In a case where the result ofthis check is negative (that is, in a case where a disconnection has notoccurred), the controller 431 releases the I/O suppression (refer toFIG. 10B), and performs the discover process thereafter. For thisreason, even when the discover process is in progress on thecommunication path that is coupled to the controller 431, thiscontroller 431 outputs an I/O command to this communication path.Alternatively, in a case where the result of the above-mentioned checkis affirmative, I/O suppression is maintained (Refer to FIG. 10A). Thediscover process is carried out while I/O suppression is beingmaintained.

FIG. 11 shows the processing flow of a process related to the Aspect 1.

This process is commenced by the controller 431 that receives the errorinformation. It is supposed below that this controller 431 is thecontroller #0, and that the communication path (the relevant path)through which the error information passed is the first communicationpath 461.

The controller #0, for example, sets I/O suppression for the firstcommunication path 461 in the memory 509 (S1101).

Next, the controller #0 targets the expander #0 n and carries out thefollowing processing. The initially targeted expander #0 n is theexpander #00 (that is, the first initial-stage expander).

-   -   (*) The controller #0 issues the RG command to the expander #0 n        (S1102).    -   (*) The controller #0 determines whether or not a normal        response (a response that denotes normal) was received with        respect to this RG command (S1103).

In a case where the result of the determination in S1103 is negative,the coupling between the expander #0 n and the next expander #0 (n+1)has been disconnected. In accordance with this, the controller #0performs the discover process in order from the first initial-stageexpander #00 to the last subsequent-stage expander without releasing theI/O suppression (S1105). After the discover process has ended for allthe expanders in the first communication path 461, the controller #0releases the I/O suppression with respect to the first communicationpath (S1106: NO, S1107).

In a case where the result of the determination in S1103 is affirmative,the controller #0 issues the RG command to the next expander #0 n (wheren=n+1) (S1102). The target is the last subsequent-stage expander, and ina case where a normal response is received with respect to the RGcommand issued to this last subsequent-stage expander, the controller #0releases the I/O suppression with respect to the first communicationpath 461 (S1104). Thereafter, the controller #0 performs the discoverprocess in order from the first initial-stage expander #00 to the lastsubsequent-stage expander (S1105).

According to the Aspect 1 above, the controller 431, upon receiving thebroadcasted error information, checks whether or not the cause of theerror information that has been broadcast is the disconnection of thecoupling between the expander #x and the next expander #(x+1) prior tocommencing the discover process. In a case where the result of thischeck is negative, the controller 431 releases the I/O suppression withrespect to the first communication path through which the errorinformation passed, and thereafter, commences the discover process. Thatis, the length of time during which the I/O suppression is set may beshortened in a case where it has been identified that releasing the I/Osuppression with respect to the communication path that carried theerror information would have little affect.

Furthermore, the issuing of the RG command may be omitted in S1105.Specifically, a discover command that specifies the phy numberidentified on the basis of the response to the RG command in S1102 maybe issued in S1105.

<Aspect 2-1: Wide Link Summary Command>

Hereinbelow, the respective phys that are coupled to the respectivephysical links that configure a wide link will be called “wide phy”.Also, a wide phy that is directly coupled to a subsequent-stage expanderwill be called a “subordinate wide phy”.

The PDEV:T is directly coupled to the expander #x. In addition, it issupposed that one or more expanders that come earlier in the initialstage than the expander #x are coupled using the wide link, and I/O ispossible to the PDEV:T through any subordinate wide phy of the two ormore subordinate wide phys that are coupled to the wide link.

According to the SAS standard, the controller 431, upon recognizing thePDEV:T anew, does not issue the CRI command for setting the SAS addressinformation “M” for each subordinate wide phy in all the expanders thatcome earlier in the initial stage than the expander #x. Specifically,the number of CRI commands to be issued is the product of (the number ofexpanders that come earlier in the initial stage than the expander #x)and (the number of subordinate wide phy that come earlier in the initialstage than the expander #x).

The same SAS address information “M” is set in the expander route table611 for two or more subordinate wide phys that are coupled to the samewide link.

Focusing on this fact, this embodiment defines the wide link bundlecommand as an original CMI command (an expanded CMI command). The CRIcommand issued in the discover process may be deemed a wide link bundlecommand as appropriate.

Specifically, for example, as shown in FIG. 12, the controller #0 issuesa wide link bundle command 1201 to the expander #00 (the expander thatcomes earlier in the initial stage than the target expander of thediscover process) as the CRI command in the discover process withrespect to the expander that comes later in the subsequent stage thanthe expander #00. The wide link bundle command 1201 comprises phyinformation, an index number and SAS address information as parameters.According to the example of FIG. 12, the phy information denotes theidentification numbers “0”, “1”, “2” and “3” of all the subordinate widephys that are coupled to the wide link. The index number, for example,is “3”. The SAS address information denotes the SAS address “X” of thenewly recognized PDEV:X.

The expander #00 receives the wide link bundle command 1201. Thefirmware 631 (the command processing program), in accordance with thewide link bundle command 1201, sets the SAS address information “X” inall the entries corresponding to the phy numbers “0”, “1”, “2” and “3”in the column (the column inside the routing table 611) of the indexnumber “3” as shown in the dotted-line box.

In this way, according to the Aspect 2-1, a single wide link bundlecommand may be used to set the same SAS address information in theexpander route table 611 for two or more subordinate wide phys that arecoupled to the same wide link.

The configuration of the wide link bundle command will be explainedbelow.

FIG. 13 shows the configuration of the wide link bundle command.

FIG. 14 shows the configuration of the ordinary CRI command.

In the following explanation, it is supposed that the CRI command isconfigured using a plurality of entries, each entry is one byte in size,each entry is configured from eight sub-entries, and each sub-entry isone bit in size. Furthermore, the f^(th) entry will be described as“entry #f” (where, according to FIGS. 13 and 14, f is an integer from 0to 43).

The configuration of the wide link bundle command is substantially thesame as that of the ordinary CRI command with the exception of thefollowing points.

-   -   (*) Entry #1. Both the wide link bundle command and the ordinary        CRI command comprise a code denoting a function in entry #1. The        function denotes the type of command. The function code of the        wide link bundle command is function code “COh”, which is        allocated to the wide link bundle command from two or more        undefined function codes of a plurality of function codes.        Specifically, for example, according to the SAS 2.0 standard,        “COh” through “FFh” are codes that may be freely used by a        vendor. In this embodiment, the “COh” is allocated to the wide        link bundle command from thereamong.    -   (*) Entry #9. The ordinary CRI command comprises a single phy        identifier (phy number) in the entry of the ninth byte.        Alternatively, the wide link bundle command must specify all of        the two or more subordinate wide phys that are coupled to the        same wide link, and, in addition, there may be times when the        size of the information denoting these two or more subordinate        wide phys exceeds one byte, and, for this reason, this command        does not comprise the phy number in a single entry #9.    -   (*) Entries #24 through #39. The ordinary CRI command does not        comprise any particularly valid information in entries #24        through #39. Alternatively, the wide link bundle command        comprises phy numbers in the entries #24 through #39. In this        embodiment, the phy information is a target phy identifier        bitmap (hereinafter, the phy bitmap). One example of the phy        bitmap is as shown in FIG. 15. The phy bitmap is configured from        a plurality of bits (128 bits) respectively corresponding to a        plurality of phys (for example, 128 phys). Within the phy        bitmap, two or more bits corresponding to two or more specified        phys (two or more subordinate wide phys that are coupled to the        same wide link) are each ON (for example, “1”), and the other        bits are OFF (for example, “0”). Furthermore, the phy        information may be another type of information beside the phy        bitmap, for example, two or more phy numbers denoting two or        more specified phys.

Both the wide link bundle command and the ordinary CRI command compriseindex numbers in entries # 6 and #7, and comprise SAS addressinformation in entries #16 through #23.

FIG. 16 shows the processing flow of a process that conforms to theAspect 2-1 (a process performed by the controller 4431). Furthermore, inthe explanation of FIG. 16, the destination expander of the CRI commandin the discover process (the expander that comes earlier in the initialstage than the expander that is the target of the discover process) willbe called the “target expander”.

The controller 431 identifies two or more subordinate wide phys coupledto the same wide link from among the plurality of phys of the targetexpander based on the device management table 511 (Refer to FIG. 8)(S1601). The device management table 511 may comprise informationdenoting which phy is the subordinate phy (the phy that is directlycoupled to the subsequent-stage expander).

The controller 431 creates a wide link bundle command (S1602).Specifically, the controller 431 sets in the CRI command a format code(COh), an index number, SAS address information, and a phy bitmap inwhich two or more bits corresponding to the two or more subordinate widephys identified in S1601 are ON.

The controller 431 issues the wide link bundle command created in S1602to the target expander (S1603).

FIG. 17 shows the processing flow of a process performed by the expander443.

The expander 443 receives the command (S1701).

The command processing program 644 inside the firmware 631 analyzes thecommand, and determines whether or not this command is a wide linkbundle command (S1702). The type of this command may be identified fromthe function code inside the received command.

In a case where the result of the determination in S1702 is negative,the program 644 carries out processing that conforms to the receivedcommand (S1706). For example, in a case where the command is an I/Ocommand, the transfer of this I/O command is carried out. Furthermore,in a case where the command is a batch delete command, all of the SASaddress information is deleted from the expander route table 611.

In a case where the result of the determination in S1702 is affirmative,the program 644 performs the following processing in order from thestart of the phy bitmap to the end.

-   -   (*) The program 644 determines whether or not the first bit of        the one or more bits for which the determination of S1702 has        not been carried out is ON (S1703).    -   (*) In a case where the result of the determination in S1703 is        affirmative (S1703: YES), the program 644 sets the SAS address        information in the expander route table 611 (S1704).        Specifically, the program 644 sets the SAS address information        inside the received wide link bundle command in the entry for        the phy number corresponding to this bit in the column        corresponding to the index number inside this command.    -   (*) Either after S1704, or in a case where the result of the        determination in S1703 is negative, the program 644 determines        whether or not the bit in S1703 is at the end. In a case where        the result of this determination is negative, the program 644        carries out S1703 for the next bit, and in a case where the        result of this determination is affirmative, the program 644        carries out S1705.

The program 644 returns a response with respect to the wide link bundlecommand to the controller 431 (S1705).

According to the Aspect 2-1 above, in the discover process, a singlewide link bundle command (an original CRI command) may be used to setthe same SAS address information in the expander route table 611 for twoor more subordinate wide phys configuring the same wide link. This makesit possible to reduce the number of CRI commands that are issued in thediscover process, and to shorten the time required for the discoverprocess accordingly.

<Aspect 2-2: Skip Invalidation>

It is supposed that a status change (for example, link-down) hasoccurred for the PDEV:K that is directly coupled to the expander #02 asshown in FIG. 18A. The expander #02 detects this status change, andbroadcasts this report (for example, error information). The controller#0 receives the report and issues discover commands in order from thefirst initial-stage expander #00. The discover command is issued foreach phy. This makes in possible for the controller #0 to detect thestatus change (for example, the link-down) with respect to the PDEV:Kthat is directly coupled to the expander #02.

In a case where the controller #0 detects a link-down with respect tothe PDEV:K, the controller #0 must issue the CRI command to all theexpanders #00 and #01 that come earlier in the initial stage than theexpander #02. This CRI command is for setting the status information“invalid” in the expander route table 611 with respect to the PDEV:Kthat is indirectly coupled to either the expander #00 or #01 that is thedestination of this command. This CRI command is issued for eachsubordinate phy that is indirectly coupled to the PDEV K. As a result ofthis, the expander route table 611 of the expander #00 (and #01), forexample, is updated as shown in the dotted-line box in FIG. 18B. Thisprocessing is carried out each time a link-down is detected with respectto a PDEV.

In addition, the controller #0 must also issue the CRI command to allthe expanders #00 and #01 that come earlier in the initial stage thanthe expander #02 even in a case where the controller #0 detects alink-up with respect to the PDEV:K. This CRI command is for setting thestatus information “valid” in the expander route table 611 with respectto the PDEV:K that is indirectly coupled to either the expander #00 or#01 that is the destination of this command. This CRI command is issuedfor each subordinate phy that is indirectly coupled to the PDEV:K. As aresult of this, the expander route table 611 of the expander #00 (and#01), for example, is updated as shown in the dotted-line box in FIG.18C. This processing is carried out each time a link-up is detected withrespect to a PDEV.

According to the explanation that referenced FIGS. 18A through 18C, theCRI command is issued each time that a link-down and a link-up isdetected.

However, there is an intermittent link-down. Also, there are times whena PDEV that detects a link-down is able to continue to be used. In casessuch as these, issuing the CRI command for setting invalid just becausea link-down was detected would be wasteful. This is because, in the end,a link-up will be detected and the CRI command for setting valid will beissued.

According to the Aspect 2-2, in a case where a link-down is detected, atleast one of the following determinations (1) and (2) is carried out,

-   -   (1) whether or not the PDEV that detected the link-down is able        to continue to be used, and    -   (2) whether or not the link-down is an intermittent link-down.

In a case where the result of at least one of these determinations isaffirmative (in this embodiment, in a case where the results of bothdeterminations are affirmative), the issuing of the CRI command forsetting invalid is skipped.

Furthermore, according to the Aspect 2-2, the CRI command for settingvalid is not issued even when a link-up is detected.

For this reason, according to the Aspect 2-2, the number of issued CRIcommands is reduced.

The Aspect 2-2 will be explained in more detail below.

FIG. 19 shows the processing flow of a process that conforms to theAspect 2-2 (a process performed by the controller 431 (in particular, bythe device command issuing program 515)). This processing flow will beexplained by giving the controller #0 as an example. Hereinafter, thePDEV that detected the link-down will be called the “target PDEV”, thephy that is directly coupled to the target PDEV will be called the“target phy”, and the expander that comprises the target phy will becalled the “target expander”.

The controller #0, upon receiving the error information (S1901: YES),issues the discover command (S1902). The destination of the discovercommand moves one at a time in order in the subsequent stage from theexpander #00. Furthermore, since the number of phy specified in thediscover command is one, the discover command is issued for each phy ofthe relevant expander #0 n with respect to each expander #0 n. Thelink-down is detected by the controller #0 in accordance with the targetphy-specifying discover command that is issued to the target expander(S1903: YES).

Furthermore, in a case where S1901 is YES, the processing described forthe Aspect 1, that is, the following processing may be carried out.

-   -   (1) Set I/O suppression with respect to the first communication        path 461 that carried the error information, and    -   (2) issue the RG command (the destination of the RG command        shifts from the expander #00 to the subsequent stage one at a        time in order). In the Aspect 2-2, link-down has simply occurred        with respect to the PDEV without the coupling between the        expander and the next expander being disconnected. For this        reason, the controller #0 may release the I/O suppression and        carry out S1902 thereafter.

In a case where S1903 is YES, the controller #0 sets “abnormal” in thedevice management table 511 (refer to FIG. 8) with respect to the targetphy (S1904).

Next, the controller #0 executes the failure check process (S1905). Inthis process, a determination as to whether or not it is possible tocontinue to use the target PDEV is made. In a case where the target PDEVlink-down count exceeds a prescribed number of times in this process, adetermination is made that the continued use of the target PDEV is notpossible. Alternatively, in a case where the target PDEV link-down countis equal to or less than the prescribed number of times, a determinationis made that the continued use of the target PDEV is possible.

In a case where the determination in S1905 is that continued use is notpossible (S1906: NO), the controller #0 issues the CRI command forsetting “invalid” with respect to the SAS address information of thetarget PDEV to each expander that comes earlier in the initial stagethan the target expander (S1909).

In a case where the determination in S1905 is that continued use ispossible (S1906: YES), the controller #0 executes a link-down checkprocess (S1907). In this process, a determination is made as to whetherthe link-down is intermittent or permanent. In this process, in a casewhere a link-up is detected with respect to the target PDEV within aprescribed time period from the time this process started, the detectedlink-down is determined to be intermittent. Alternatively, in a casewhere a link-up is not detected with respect to the target PDEV eventhough a prescribed time period has elapsed from the time this processstarted, the detected link-down is determined to be permanent.

In a case where the determination in S1907 is permanent (S1908: NO), thecontroller #0 carries out the above-mentioned S1909.

In a case where the determination in S1907 is intermittent (S1908: YES),the controller #0 skips S1909.

FIG. 20 shows the processing flow of the failure check process (S1905 ofFIG. 19).

The controller #0 (the device command issuing program 515) increments byone the link-down count with respect to the target PDEV (S2001).Specifically, the link-down count table 512 (refer to FIG. 5) comprisesSAS address information and a count value denoting the link-down countfor each PDEV. The program 515 increments by one the count valuecorresponding to the SAS address information of the target PDEV.

The program 515 determines whether the post-update link-down countexceeds a prescribed number of times (S2002).

In a case where the result of the determination in S2002 is affirmative(S2002: YES), the program 515 determines that the continue use of thetarget PDEV is not possible (S2003).

In a case where the result of the determination in S2002 is negative(S2002: NO), the program 515 determines that the continue use of thetarget PDEV is possible (S2004).

FIG. 21 shows the processing flow of the link-down check process (S1907of FIG. 19).

The controller #0 (device command issuing program 515) registersinformation denoting the current time (the time acquired from the timer505) with respect to the target PDEV in the initial save time table 510(S2101). Specifically, the initial save time table 510 (refer to FIG. 5)comprises information denoting the SAS address information and time atwhich the link-down check process started for each PDEV. The program 515registers information denoting the timer 505-acquired time at which thelink-down check process started in the entry that corresponds to the SASaddress information of the target PDEV. In the explanation of FIG. 21below, the time denoted by the information registered in the entry ofthe table 510 with respect to the target PDEV will be referred to as the“initial time”.

The program 515 waits for a fixed period of time after S2101 (S2102),and issues a discover command specifying the target phy to the targetexpander (S2103).

In a case where a link-up is detected on the basis of the response tothe discover command issued in S2103 (S2104: YES), the program 515determines that the link-down is intermittent (S2108).

In a case where a link-up is not detected on the basis of the responseto the discover command issued in S2103 (in a case where a link-down isdetected) (S2104: NO), the program 515 acquires the current time anddetermines whether or not a prescribed period of time has elapsed(S2106). Specifically, the program 515 determines whether or not thedifference between this current time and the initial time is equal to orlarger than a fixed time period.

In a case where the result of the determination in S2106 is affirmative(S2106: YES), the program 515 determines that the link-down is permanent(S2107).

In a case where the result of the determination in S2106 is negative(S2106: NO), the program 515 waits for a fixed period of time (S2102),and once again issues the discover command specifying the target phy tothe target expander (S2103).

According to the Aspect 2-2 above, the number of issued CRI commands canbe reduced.

<Aspect 2-3: Batch Delete Command>

It is unclear what SAS address information is set in the expander routetable 611 of an expander newly detected via the discover process. Forthis reason, unless all of the SAS address information is deleted fromthe expander route table 611 temporarily and the discover process is notcarried out in order from the first initial-stage expander, there may beinstances where the correct information is not set in the expander routetable 611. This makes it impossible to access a specific device.

However, when an ordinary CRI command is used to delete all the SASaddress information from the expander route table 611, a CRI commandmust be issued for each entry. For this reason, the CRI command issuecount becomes enormous.

Accordingly, in the Aspect 2-3, all of the SAS address information isdeleted from the expander route table 611 using a single command. Thismakes it possible to reduce the number of CRI commands.

The Aspect 2-3 will be explained in detail below. Furthermore, the“deletion of SAS address information” may involve setting “invalid” inthe entry comprising the SAS address information (that is, substantiallydeleting the SAS address information) instead of actually deleting theSAS address information.

Examples of cases in which failure to invalidate the entire expanderroute table 611 at one time makes it impossible to access a specificdevice, for example, are shown in FIGS. 22A through 24B.

As shown in FIG. 22A, the expander #011 is directly coupled to the phy#1 of the expander #00, and the expander #012 is directly coupled to thephy #2 of the expander #00. The PDEV:A, the PDEV:B and the PDEV:C aredirectly coupled to the expander #011, and the PDEV:X and the PDEV:Y aredirectly coupled to the expander #012. For this reason, the expanderroute table 611 of the expander #00 shows that the PDEV:A, the PDEV:Band the PDEV:C are indirectly coupled to the phy #1, and the PDEV:X andthe PDEV:Y are indirectly coupled to the phy #2 as shown in FIG. 22A.

It is supposed that the following is carried out at this point.

-   -   (1) As indicated by reference sign 2211 in FIG. 22B, the        coupling between the controller #0 and the expander #00 is        disconnected (for example, the cable has been removed).    -   (2) Next, as indicated by reference sign 2301 in FIG. 23A, the        expander #012 is directly coupled to the phy #1 of the expander        #00 instead of to the expander # 011, and, in addition, expander        #011 is directly coupled to the phy #2 of the expander #00        instead of to the expander # 011.    -   (3) Finally, as indicated by reference sign 2311 in FIG. 23B,        the SAS controller #0 and the expander #00 are coupled together        once again (for example, the cable is connected).

Thereafter, the controller #0 carries out the discover process in orderfrom the expander #00 to the subsequent-stage expander as shown in FIG.24A (S2401).

The number of pieces of new SAS address information (X, Y) here islarger than the number of pieces of old SAS address information (A, B,C) with respect to the phy #2. For this reason, all the old SAS addressinformation is updated to the new SAS address information, andtherefore, the old SAS address information does not remain in the table611 with respect to the phy #2.

However, the number of pieces of new SAS address information (X, Y) issmaller than the number of pieces of old SAS address information (A, B,C) with respect to the phy #1. For this reason, as indicated byreference sign 2402 in FIG. 24A, the old SAS address information “C”,which is not updated by the new SAS address information, remains in thetable 611 with respect to the phy #1. As a result of this, the old SASaddress information “C” remaining with respect to the phy #1 overlapswith the new SAS address information “C” for the phy #2.

In a case where there are a plurality of phys that are indirectlycoupled to the target (the SAS address information) of the receivedcommand for each expander, the command is transferred from the phyhaving the smallest phy number of these plurality of phys. In the table611, the SAS address information “C” is set in both phys #1 and #2. Forthis reason, in a case where the expander #00 receives a command that istargeted at the PDEV:C, the command is transferred from the phy #1despite the fact that the PDEV:C is not coupled to the phy #1. However,since the PDEV:C is not coupled to the command-destination expander#012, even when the expander #012 receives this command, as shown inFIG. 24B, the transfer-destination of this command is unable to bedetected.

Accordingly, as shown in FIG. 25A, the controller #0 (the device commandissuing program 515) issues a batch delete command to the newly detectedexpander #00, and the expander #00 (the command processing program 644)collectively deletes all of the SAS address information, from theexpander route table 611 of the expander #00 in accordance with thisbatch delete command.

Thereafter, as shown in FIG. 25B, the discover process is carried out inorder from the expander #00 to the subsequent-stage expanders (S2511).In accordance with this, as shown in FIG. 25B, the correct SAS addressinformation is set in the phy #1 and #2 in the expander route table 611of the expander #00. That is, as shown in FIGS. 24A and 24B, the SASaddress information does not inadvertently overlap.

As described hereinabove, one example of a case in which it can becomenecessary to delete all the SAS address information from the expanderroute table 611 of a newly detected expander is one such as that shownin FIG. 26. According to FIG. 26, there are two or more subsequent-stageexpanders directly coupled to a single expander. Specifically, thestorage system comprises a switchbox 2601 between chassis 413 andchassis 413 (or, between chassis 411 and chassis 413). The switchbox2601 comprises duplicated expanders (#01, #11) (Hereinafter, theexpander inside the switchbox 2601 will be referred to as the “SWexpander”.). The one SW expander #01 is directly coupled to two or moreof the one expanders #021 and #022 inside two or more additional chassis413, and the other SW expander #11 is directly coupled to two or more ofthe other expanders #121 and #122 inside two or more additional chassis413.

According to the configuration shown in FIG. 26, the above-mentionedproblem of being unable to detect the transfer destination may occur.For example, in a case where the expander directly coupled to the firstphy of the SW expander #01 changes from the expander #021 to theexpander #022, and, in addition, the expander directly coupled to thesecond phy of the SW expander #01 changes from the expander #022 to theexpander #021, the problem of not being able to detect the transferdestination can occur.

FIG. 27 shows the issuing of the batch delete command and the batchdelete of SAS address information (FIG. 27 uses controller #0 as anexample).

The controller #0 issues the batch delete command 2701 to a newlydetected expander #00. The expander #00 deletes all of the SAS addressinformation from the expander route table 611 in accordance with thiscommand 2701.

FIG. 28 shows the configuration of the batch delete command.

The batch delete command is an original CRI command (an expanded CRIcommand). According to the standard SAS 2.0, “COh” through “FFh” arecodes that the vendor can make free use of as described hereinabove. Inthis embodiment, “C1h” from thereamong is allocated to the batch deletecommand as the function code. The expander 443, upon detecting that thefunction code inside the received CRI command is “C1h”, identifies thisCRI command as the batch delete command, and deletes all the SAS addressinformation from the expander route table 611.

FIG. 29 shows the processing flow of a process (a process carried out bythe controller 431 (in particular, the device command issuing program515)) that conforms to the Aspect 2-3.

The program 515 issues the discover command to the first initial-stageexpander of the expanders not targeted in S2902 (S2902). The program 515determines whether or not a new expander has been detected on the basisof the response with respect to this discover command (S2903). Thisdetermination is carried out, for example, on the basis of themanagement information of the controller 431. As used here, “managementinformation”, for example, comprises the corresponding relationshipbetween the phy number and the SAS address information, which isdirectly coupled to this phy, for each expander. Therefore, a case inwhich the (p) and the (q) below differ from one another signifies that anew expander has been detected (Hereinafter, an expander that has beendetected anew will be referred to as the “newly detected expander”.).

-   -   (p) The SAS address information of the response with respect to        the discover command.    -   (q) The management-information SAS address information        associated with the phy number specified in the discover        command.

The target of the discover command changes from an initial-stageexpander to a subsequent-stage expander, but in a case where a newexpander is first detected, all the expanders that come later in thesubsequent stage than this newly detected expander are detected anewthereafter.

In a case where the result of the determination in S2903 is negative(S2903: NO), the program 515 determines whether or not the recent targetin S2902 is the last subsequent-stage expander. In a case where theresult of this determination is negative, the expander after the targetin the recent S2902 is made the target, and S2902 is carried out.

Alternatively, in a case where the result of the determination in S2903is affirmative (S2903: YES), the following processing is carried out.

-   -   (S2904) The program 515 issues the CRI command to all the        expanders that come earlier in the initial stage than the target        in S2902. This CRI command comprises SAS address information of        a newly detected expander, an index number, and a phy number        that has been specified in the discover command of S2902. The        expander that receives this CRI command sets the SAS address        information inside the CRI command in the entry corresponding to        the phy number and the index number inside the CRI command in        the expander route table 611.    -   (S2905) The program 515 issues the batch delete command to the        newly detected expander. The newly detected expander receives        the batch delete command, and deletes all of the SAS address        information from the expander route table 611 of this expander.    -   (S2906) The program 515 executes the discover process for the        newly detected expander. In accordance with this, the SAS        address information of the device directly coupled to the newly        detected expander is set in the expander route tables 611 of all        the expanders that come earlier in the initial stage than the        newly detected expander.

An embodiment of the present invention has been explained hereinabove,but this embodiment is simply an example for explaining the presentinvention, and does not purport to limit the scope of the presentinvention solely to this embodiment. The present invention can be putinto practice using various other modes as well.

REFERENCE SIGNS LIST

-   401 Storage system-   431 Controller-   443 Expander

1. A storage system comprising: a plurality of storage devices; firstand second controllers coupled to a host; a first communication path,which is coupled to the first controller and the plurality of storagedevices, and which is a path for communications according to SAS; and asecond communication path, which is coupled to the second controller andthe plurality of storage devices, and which is a path for communicationsaccording to SAS, wherein each communication path has a plurality ofexpanders that are coupled in series, and a first initial-stage expanderof these plurality of expanders is coupled to the controller, whereineach expander has a plurality of phys, and these plurality of physinclude a phy to which the storage device is coupled, and a phy to whichthe expander is coupled, wherein the first controller (A) sets I/Osuppression for the first communication path, (B) determines, subsequentto the (A), whether or not the coupling between the first controller andthe expander, or the coupling between the expanders has beendisconnected, (C) in a case where a result of the determination in the(B) is affirmative, without releasing the I/O suppression executes withrespect to at least one expander in the first communication path aninvestigation process that includes a process that issues a command foracquiring information related to a device that is directly coupled tothis expander, and releases the I/O suppression thereafter, and (D) in acase where the result of the determination in the (B) is negative,executes the investigation process for at least one expander in thefirst communication path after releasing the I/O suppression, andwherein while the first communication path is under I/O suppression, thefirst and second controllers issue I/O commands targeted at the storagedevice by way of the second communication path.
 2. A storage systemaccording to claim 1, wherein the first controller in the (B) (b1)issues an RG (Report General) command to the first initial-stageexpander in the first communication path, (b2) determines whether aresponse has been normally received with respect to the issued RGcommand, and (b3) in a case where the result of the determination in the(b2) is affirmative, issues the RG command to the first initial-stageexpander of the one or more expanders that have not as yet been thedestination of the RG command, and thereafter, carries out the (b2),wherein, when the result of the determination in the (b2) is negative,the result of the determination in the (B) is affirmative, and wherein,when the result of the determination in the (b2) is affirmative for allthe expanders in the first communication path, the result of thedetermination in the (B) is negative.
 3. A storage system according toclaim 2, wherein each expander comprises routing control informationthat is used in routing, and the routing control information cancomprise SAS address information denoting the SAS address of a devicethat is indirectly coupled to each phy, (X) wherein the firstcontroller, in the investigation process, issues a first informationsetting command, which includes phy information and SAS addressinformation, to the expander in the communication path coupled to thiscontroller, (Y) wherein the expander that has received the firstinformation setting command sets, in the routing control information ofthis expander, the SAS address information inside this first informationsetting command, for the phy that is identified from the phy informationin this first information setting command, wherein each of thecontrollers comprises device management information that denotes whichphy of which expander in the communication path coupled to thiscontroller is coupled to a physical link inside a wide link, wherein, inthe (X), the first controller identifies, based on the device managementinformation, two or more phys, which are coupled to two or more physicallinks configuring the same wide link, from among the plurality of physof the expander that is the issue-destination of the first informationsetting command, wherein the phy information includes informationdenoting the identified two or more phys, and wherein, in the (Y), theexpander sets, in the routing control information, the SAS addressinformation inside the information setting command for all of the two ormore phys identified from the phy information inside the receivedinformation setting command.
 4. A storage system according to claim 3,wherein the first information setting command is a wide link bundlecommand, wherein the wide link bundle command is a command that conformsto SMP (Serial Management Protocol), and is a CRI (Configure RouteInformation) command, in which a specific code is described in aprescribed location, wherein the specific code is a function code thathas been allocated to the wide link bundle command from two or moreundefined function codes of a plurality of function codes, wherein thephy information is described, in the CRI command, in a location in placeof a location in which the phy identification information is set,wherein the phy information is a bitmap, this bitmap is configured froma plurality of bits respectively corresponding to a plurality of phys,and in this bitmap, the two or more bits that respectively correspond tothe two or more phys are ON, and wherein the expander that receives theCRI command performs the (Y) in a case where the function code, which isdescribed in the prescribed location of this CRI command, is detected tobe the prescribed code.
 5. A storage system according to claim 2,wherein each expander comprises routing control information that is usedin routing, wherein this routing control information can include SASaddress information denoting an SAS address of a device indirectlycoupled to each phy, and status information denoting that this storagedevice is either valid or invalid, wherein, in the (D), (d1) the firstcontroller, in the investigation process, detects a link-down for atarget device, which is a storage device that is directly coupled to acertain phy of a certain expander in the first communication path, (d2)the first controller determines whether or not this link-down satisfiesa prescribed condition, (d3) in a case where the result of thedetermination in the (d2) is negative, the first controller issues asecond information setting command, which includes information denotingthat the target device is invalid, and phy information denoting a phythat is indirectly coupled to this target device, to each expander thatexists in the initial stage further forward than the certain expander,and the expander, which has received this second information settingcommand, sets status information denoting that the target device isinvalid in the routing control information for the phy identified fromthe phy information inside this second information setting command, and(d4) in a case where the result of the determination in the (d2) isaffirmative, the first controller does not issue the second informationsetting command to each expander that exist on the target controllerside of the certain expander.
 6. A storage system, comprising: aplurality of storage devices; first and second controllers coupled to ahost; a first communication path, which is coupled to the firstcontroller and the plurality of storage devices, and which is a path forcommunications according to SAS; and a second communication path, whichis coupled to the second controller and the plurality of storagedevices, and which is a path for communications according to SAS,wherein each communication path comprises a plurality of expanders thatare coupled in series, a first initial-stage expander of these pluralityof expanders is coupled to the controller, physical links establishcoupling between the controller and the expander, and between theexpanders, and, in addition, at least one expander is coupled to eitherthe controller or the expander using a wide link comprising two or morephysical links, wherein each expander comprises a plurality of phys, andthese plurality of phys include a phy to which the storage device iscoupled, and a phy to which the expander is coupled, wherein eachexpander comprises routing control information that is used in routing,wherein the routing control information can include SAS addressinformation denoting the SAS address of the device that is indirectlycoupled to each phy, (X) wherein each controller issues a firstinformation setting command, which includes phy information and SASaddress information, to the expander in the communication path coupledto this controller, (Y) wherein the expander that receives the firstinformation setting command sets, in the routing control information ofthis expander, the SAS address information inside this first informationsetting command for the phy that is identified from the phy informationin this first information setting command, wherein each of thecontrollers includes device management information that denotes whichphy of which expander in the communication path coupled to thiscontroller is coupled to a physical link inside the wide link, wherein,in the (X), each of the controllers identifies, based on the devicemanagement information, two or more phys coupled to two or more physicallinks configuring the same wide link from among the plurality of phys ofthe expander that is the issue-destination of the first informationsetting command, wherein the phy information comprises informationdenoting the identified two or more phys, and wherein, in the (Y), theexpander sets, in the routing control information, the SAS addressinformation inside this information setting command for all of the twoor more phys identified from the phy information inside the receivedinformation setting command.
 7. A storage system according to claim 6,wherein the first information setting command is a wide link bundlecommand, wherein the wide link bundle command is a command that conformsto SMP (Serial Management Protocol), and is a CRI (Configure RouteInformation) command, in which a specific code is described in aprescribed location, wherein the specific code is a function code thathas been allocated to the wide link bundle command from two or moreundefined function codes of a plurality of function codes, wherein thephy information is described, in the CRI command, in a location in placeof a location in which the phy identification information is set,wherein the phy information is a bitmap, this bitmap is configured froma plurality of bits respectively corresponding to a plurality of phys,and in this bitmap, the two or more bits that respectively correspond tothe two or more phys are ON, and wherein the expander that receives theCRI command performs the (Y) in a case where the function code, which isdescribed in the prescribed location of this CRI command, is detected tobe the prescribed code.
 8. A storage system, comprising: a plurality ofstorage devices; first and second controllers coupled to a host; a firstcommunication path, which is coupled to the first controller and theplurality of storage devices, and which is a path for communicationsaccording to SAS; and a second communication path, which is coupled tothe second controller and the plurality of storage devices, and which isa path for communications according to SAS, wherein each communicationpath comprises a plurality of expanders that are coupled in series, anda first initial-stage expander of these plurality of expanders iscoupled to the controller, wherein each expander comprises a pluralityof phys, and these plurality of phys include a phy to which the storagedevice is coupled, and a phy to which the expander is coupled, whereineach expander includes routing control information that is used inrouting, wherein the routing control information can include SAS addressinformation denoting the SAS address of the device that is indirectlycoupled to each phy, and status information denoting whether thisstorage device is valid or invalid, wherein in a case where either thefirst or the second controller detects a link-down for a target device,which is the storage device that is directly coupled to a certain phy ofa certain expander in the communication path that is coupled to thiscontroller, (A) the target controller, which is the controller that hasdetected this link-down, determines whether or not this link-downsatisfies a prescribed condition, (B) in a case where the result of thedetermination in the (A) is negative, the target controller issues asecond information setting command, which includes information denotingthat the target device is invalid and phy information denoting the phythat is indirectly coupled to the target device, to each expander thatexists in the initial stage further forward than the certain expander,and the expander, which has received the second information settingcommand, sets, in the routing control information, the statusinformation denoting that the target device is invalid for the phyidentified from the phy information inside this second informationsetting command, and (C) in a case where the result of the determinationin the (A) is affirmative, the target controller does not issue thesecond information setting command from the certain expander to eachexpander that exists on the target controller side.
 9. A storage systemaccording to claim 8, wherein, in the (A), the target controller (a01)updates the number of times that link-down has been detected for thetarget device, and (a02) determines whether or not a post-update numberof times is less than a prescribed number of times, and in a case wherethe result of the determination in the (a02) is negative, the result ofthe determination in the (A) is negative.
 10. A storage system accordingto claim 9, wherein, in a case where the result of the determination inthe (a02) is negative, in the (A) the target controller (a11) checkswhether or not link-up has occurred with respect to the target devicewithin a specified time from the time that link-down has been detected,and wherein, in a case where the result of the check in the (a11) isnegative, the result of the determination in the (A) is negative, andwherein, in a case where the result of the check in the (a11) isaffirmative, the result of the determination in the (A) is affirmative.11. A storage system according to claim 8, wherein, in the (A) thetarget controller (a11) checks whether or not link-up has occurred withrespect to the target device within a specified time from the time thatlink-down has been detected, and in a case where the result of the checkin the (a11) is negative, the result of the determination in the (A) isnegative.
 12. A storage system, comprising: a plurality of storagedevices; first and second controller coupled to a host; a firstcommunication path, which is coupled to the first controller and theplurality of storage devices, and which is a path for communicationsaccording to SAS; and a second communication path, which is coupled tothe second controller and the plurality of storage devices, and which isa path for communications according to SAS, wherein each communicationpath comprises a plurality of expanders that are coupled in series, anda first initial-stage expander of these plurality of expanders iscoupled to the controller, wherein each expander comprises a pluralityof phys, and these plurality of phys include a phy to which the storagedevice is coupled, and a phy to which the expander is coupled, whereineach expander comprises routing control information that is used inrouting, wherein the routing control information can include SAS addressinformation denoting the SAS address of the device that is indirectlycoupled to each phy, (P) wherein, in a case where the coupling betweenthe controller and the expander is disconnected and the coupling of theexpander to this controller is detected, each of the controllers issuesa batch delete command, which is a command for deleting all the SASaddress information from the routing control information, to the coupledexpander, and (Q) wherein, this expander receives this batch deletecommand, and deletes all the SAS address information from the routingcontrol information.
 13. A storage system according to claim 12, whereinthe batch delete command is a command that conforms to SMP (SerialManagement Protocol) and is a CRI (Configure Route Information) commandin which a specified code is described in a prescribed location, whereinthe specified code is a function code, which has been allocated to thebatch delete command from two or more undefined function codes of aplurality of function codes, and wherein, in the (Q), the expanderdeletes all SAS address information from the routing control informationin a case where the function code described in the prescribed locationof the CRI command is the specified code.
 14. A storage system accordingto claim 13, wherein, in each communication path, a first expander iscoupled to a first phy of one expander, a second expander is coupled toa second phy of this one expander, and the first and second expandersexist in a stage subsequent to the one expander, and wherein, after thecoupling between the first or the second controller and the expander hasbeen disconnected, the second expander is coupled to the first phy inplace of the first expander and the first expander is coupled to thesecond phy in place of the second expander.